The system generally relates to workflow processing and, more particularly to optimizing workflow execution by making intelligent decisions regarding how operations used by the workflow can be provided using modules that are dynamically loaded and continuously optimized in response to changes in system and network resource availability.
Workflow processing technologies related to big data systems, including Apache MapReduce, Apache Storm, and Apache Spark, and the like, are typically defined by a set of processing operations, enabling these operations to be run in a defined sequence on a single processing device or a grid of processing devices to accomplish a processing goal. Workflow implementations are typically driven from a workflow definition that primarily focuses on moving data from one operation to another. Workflow operations will typically be managed centrally by a workflow management platform such as Apache Yarn, Apache Mesos, or the like, and be optimized across one or more systems on a compute grid and share a common data store. This implementation works well as long as the systems are in close proximity and have predictable network availability, throughput, and system resources including memory, storage, and processing.
A disadvantage with this workflow processing approach arises when grid processing devices are not in close proximity, have limited and variable network links between each processing system, and/or have rapid changes to system resources available for workflow processing. Because of this, it is challenging or impossible for these systems to utilize a common data store or to optimize workflows that span multiple processing devices. This challenge is even more pronounced when the bulk data originates at the processing devices and that data must be reduced so that it can be transmitted over network links to be utilized by other compute devices on the grid and/or archived centrally. This illustrates the big-data, little-networks problem experienced by highly distributed computing systems, particularly those involved with remote sensing.
For example, a fleet of aircraft, each carrying a LIDAR sensor and a camera, collect geospatial collections. Limitations in quantity of devices, quality of sensor data transmitted, usability of data, latency of data transmission, changes in priorities, or changing network conditions may limit algorithmic exploitation of those collections, especially at the time of collection.
Traditionally this has been solved by static deployment of specially developed or ported code. These capabilities are encumbered by long development cycles, complete software rewrites, and quality assurance processes. This approach is slow, expensive, inflexible, and does not respond to changing resource availability. This approach also effectively puts the data users at the end of data collection workflows instead of participating during the data collection process, consequently slowing down their ability to react as information needs change. With the explosion of the number and complexity of remote sensing devices, many of which are connected to the Internet of Things, this challenge is becoming more widespread.
By managing resource utilization at the module level, existing workflow engines can, without modification, execute workflows with significant improvements to responsiveness across a number of workflows competing for similar resources, particularly when those resources cannot be reallocated to other processing devices on a grid, such as processing camera data from a camera attached to a processing device.